Method and apparatus for automatic electronic replacement of billboards in a video image

ABSTRACT

Apparatus for automatic electronic replacement of a billboard in a video image including an automatic camera orientation measurement apparatus including motion measurement means operative to measure the Field of View (FOV) of the TV camera relative to a known reference position.

This Application is a continuation of U.S. provisional Application Ser.No. 08/776,038 filed May 13, 1997. entitled “Method and Apparatus ForAutomatic Electronic Replacement of Billboards In A Video Image,” nowU.S. Pat. No. 6,208,386 which was filed on Sep. 9, 1996 as InternationalApplication No. PCT/GB96/02226 and was a filing in the United Statesunder 35 U.S.C. §371 and claims the benefit under 35 U.S.C. §119 of thefiling date of this application and the benefit of the filing date ofUnited Kingdom Applications 9518439.6 and 9601101.0 filed on Sep. 8,1995 and Jan. 19, 1996 respectively.

FIELD OF THE INVENTION

The present invention relates to a method and apparatus forautomatically replacing billboards in a vide image.

BACKGROUND OF THE INVENTION

The present invention has particular use in electronic replacement ofbillboards in a stadium or other venue but can be used to provideaccurate data relating to camera orientation for other purposes.

In previous systems it has been proposed to electronically replacebillboards in a stadium which are viewed by a viewer on television. Thebillboards in the stadium are televised by a TV camera and the boardsare electronically altered so that the TV viewer at home sees adifferent board to the spectator in the stadium or other venue.

The known systems such as those described in U.S. Pat. No. 5,266,933, anapparatus and method for electronically altering video images isdisclosed. The apparatus and method disclosed in the U.S. patent andalso in U.S. Pat. No. 5,353,392 whilst theoretically allowingreplacement of billboards do not solve the many practical problemsencountered in real environments. Most of these problems are related tothe recognition and replacement processes.

Relying entirely on pattern recognition techniques which utilize onlythe video signal to identify and localise billboards for replacementintroduces major problems which affect the practical value of such asystem.

Clearly, any pattern recognition scheme, including those described inU.S. Pat. No. 5,264,933 and U.S. Pat. No. 5,353,392 must rely on usefulvisible features in the image that can be compared with pre-defineddescriptions. Such features should be located inside the billboard or atits neighbourhood.

In realistic situations, the visibility of these features might change,continuously or otherwise from practically zero to a some thresholdvisibility which allows the pattern recognition scheme to work properly.These changes can occur in the direction of growing or reducingvisibility.

Such situations include:

Acceleration or de-acceleration of camera motion introducing a hugeamount of blur.

Excessive zooming-in or zooming-out of the billboard.

Excessive occlusion by players.

Entering or exiting a camera's field of view by any combination of pan,tilt and zoom operations.

Any combination of the above mentioned mechanisms.

Therefore, in practical situations, a continuous replacement ofbillboards, is not possible. Even if an interrupted replacement wasallowed, it would require a delay of at least a few seconds to decidewhether the resulting replacement interval is acceptable or not. Such adelay is usually not permitted in live broadcasting of sports events.

Replacing arbitrary billboards introduces further problems. A seamlessreplacement requires to identify the foreground objects occluding thebillboard in order to inhibit replacement at places of occlusion.Foreground objects mainly consist of players but also the ball or otherobjects. Consider now a player with a red shirt, occluding a part of asimilarly red portion of a billboard. Colour contrast cannot be usedrobustly to identify occlusion. Furthermore, since the player is anon-rigid object, motion or shape information cannot be used accuratelyenough to guarantee perfect replacement.

Another problem which may arise in practical situation is resolution ofbillboard identity. Consider two identical billboards positioned at twodifferent locations in the arena. Suppose different replacementbillboards are assigned to each of these physical billboards, then onemust be able to tell which one is which. This can prove to be extremelydifficult especially if no unambiguous features are visible.

SUMMARY OF THE INVENTION

This invention described a robust system foe billboard replacement,based on some or all of the following key elements:

Pan, tilt, zoom and focus sensors attached to the camera, which enableafter a proper set up procedure to estimate the presence and location ofbillboards in any given video field.

Image processing methods and their embodiment which enable to refine thesensors' estimates.

Physical billboards which are coloured properly to enable the efficientdetection of occlusion by chroma-key techniques.

Colour variation or a pattern within the physical billboard for furtherenhancing the performance of the image processing methods.

The present invention has a first object to provide a method andapparatus which enables identification of the location of a billboard orother static object in a stadium or other venue in any weatherconditions with any panning speed of the camera and with any otherchange in camera parameters.

The present invention therefore provides apparatus for automaticelectronic replacement of a billboard in a video image including anautomatic camera orientation measurement apparatus including motionmeasurement means operative to measure the Field of View (FOV) of the TVcamera relative to a known reference position.

The present invention also preferably provides apparatus for automaticelectronic replacement of a billboard in a video image, including imageprocessing means for processing video signals generated by the TVcamera, in which said processing means includes calibration means forperiodically automatically calibrating the motion measurement means,apparatus in which the motion measurement means includes means formeasuring the pan tilt, zoom or focus of the camera relative to knownreference positions and apparatus in which the motion measurement meansincludes means for measuring the pan tilt, zoom and focus of the camerarelative to known reference positions.

The present invention therefore uses dynamic recalibration to correctfor residual sensor errors or abberations in an imperfect model and forsensor drift over time. Thus it is possible in accordance with thepresent invention to use less stable sensors and the apparatus andmethod in accordance with the present invention can accommodate movementin the camera position. The image correction process for calibration ofthe sensors eliminates the necessity to keep the sensors stable bymechanical means by recalibration automatically with reference to thevideo image.

In the initial set up procedure corrections can be incorporated forcalibration for billboards which are, for example, not in the centre ofthe Field of View—for example a billboard which is in the top left-handcorner of the screen can be adjusted by, for example, 3 pixels to takeinto account abberations in the camera.

Further problems which arise in the prior art systems are firstly whenthe billboard is either substantially totally occluded or secondly, isoccluded by an object, such as a player, of the same colour as the realsign on the billboard.

This can firstly, as explained above, lead to non-recognition of thebillboard and also secondly, to difficulty in satisfactory replacementof the billboard.

In the first case the real billboard can have already been replaced inthe video image but if the camera zooms into a close up or if adifferent camera is used for the close up then the lock may be lost dueto only a very small portion of the billboard being in view. In thesecond case the player may have on a strip which is the same colour asthe billboard. The prior art systems propose to distinguish thebillboard from the player on the basis of movement if the colours arethe same and to analyse the “moving” pixels to determine occlusion. Thisis reasonable in theory but fails in practice since not all payers aremoving at all times. Thus, if a number of players move in front of abillboard and one player remains after the others have moved on, theelectronics will not be able to distinguish on movement grounds. Sincecolours are distorted by floodlighting, shadows, differences inreflectivity and different lighting conditions for a foreground playerand a background billboard, there will be occasions in practice wherethe system fails. In such cases either the original billboard mayre-appear on the video image or the replacement billboard will not beaccurately occluded.

It is again possible to introduce a delay in the video transmission toenable the electronic signal processing to be more accurate but thisdoes not solve the practical problem where a plurality of players movein different directions to occlude a billboard. The necessary delay isconsidered unacceptable and will in any case not solve all of the aboveproblems.

In accordance with a preferred embodiment of the present invention it isproposed to replace the real billboards with chroma-key panels or withdelineated areas forming chroma-key panels.

Chroma-key is essentially an occlusion technique allowing, for example,a news reader to stand and move about in front of a chroma-key board,usually coloured blue or another suitable colour. The news reader(foreground) is distinguished from the chroma-key board (background) bycolour differentiation and can thus move in front of the replacementbackground with normal occlusion of the foreground and background. Thistechnique is very well known in television studio systems and isdescribed in numerous US patents, including U.S. Pat. Nos. 2,974,190 and4,200,980.

Recently, several systems which combine camera sensors with chroma-keyfor the purpose of coordinating the movement of the graphics backgroundswith those of the camera, have been described and demonstrated.

(Ref. K. Haseba et al., Real-timing compositing system-of a real cameraimage and a computer graphic image, International BroadcastingConvention, Sep. 16-20 1994, Conference publication No. 397, IEE 1994,pp. 656-660).

In principle, such an arrangement could be used for billboardreplacement where the sensors unambiguously solve the recognitionproblem and the chroma-key billboard helps to handle occlusion properly.However due to some major differences, this arrangement should beenhanced. These enhancements are the basis of the present invention.

In a virtual set application, the camera is typically 2-10 metres awayfrom the foreground and the entire field of view is usually replaced. Incomparison, a billboard may be several hundred metres from the cameraand therefore a replacement system using sensors is much moresusceptible to sensor errors:

Due to the large focal distances, the same sensor accuracy willtranslate to larger geometric registration errors.

Consider a rotary encoder of 81000 pluses/revolution, then the angularprecision is 0.0044 degrees or 75 micro-radians. The repeatability istwice as bad. Consider a shooting range of 100 m with a field of view of4 metres, then the FOV is 40 milli-radian. The error translates to 768*?150/20000=2.88 pixels.

Since the field of view includes many stationary objects (includingbillboards) which are not replaced, the human observer will be much moresensitive to the registration errors. Additional errors may originatefrom lens distortion, rotation axis which does not pass through thefocal point, non-zero roll angle, etc.

Chroma-key is basically a technique for studios where the illuminationis carefully designed and controlled and the controls of thechroma-keyer are carefully adjusted for the specific arrangement ofblue-screen colour and illumination.

In a sports event, the conditions may be highly non-ideal and requiresome modification to the chroma-key algorithms. In particular, the keyerparameters should be adapted to the specific billboards being replaceddue to changes in illumination across the arena.

Accordingly in the present invention it is proposed to use chroma-keypanels and to replace these in the video image by the replacementbillboards.

Since it is necessary for perfect occlusion that the players or otheroccluding objects are of different colour to the chroma-key panels, itis proposed in a further preferred embodiment to provide chroma-keypanels in which the colour of the panel may be changed, for example, byusing a rotating billboard structure which is known in the art. One sidecould, for example, be blue and another green. Green may be preferablein a sports environment since players tend not to wear green as thiswould not contrast with the background sports surface.

In a further preferred embodiment and in particular where a plurality ofbillboards require replacement, a patterned chroma-key board is used.The pattern may be of any suitable shape but is preferably selected tobe suitable for the size and shape of the billboard or series ofbillboards and also to the anticipated video conditions. Thus if abillboard is only able to be viewed from a long distance then adifferent pattern will be selected to a billboard which is to be viewedin close up

The pattern may comprise different colours or may be different shades ofthe same colour. The pattern may comprise vertical and horizontal linesor may comprise a decorative pattern, a discernible advertisement,company logo or other suitable wording which may be more aestheticallyacceptable.

The use of a pattern allows further discrimination of the position ofthe camera and may allow movement of the camera from a fixed position.

The camera orientation data can be transmitted together with the videosignal and will identify the position of the billboard in any weather,lighting or occlusion conditions. No reference is required to anyfeature within the sports venue to identify the position of thebillboard.

The camera sensors can be accurate to a few pixels or in physical termsto approximately 1 cm at a range of about 100 metres thereby enablingaccurate replacement of any billboard. The recalibration can be carriedout continuously or only periodically, particularly if an initialadjustment of the calibration of billboards not in the centre of the FOVis recorded on set up.

By use of the chroma-key techniques there is no requirement to transmitany occlusion data since this can be readily inserted at a receiver andthe occlusion inserted in the normal manner.

In a preferred arrangement within a stadium or other sports venue realbillboards with normal advertising material will be situated on one sideof the stadium to be viewed by a first plurality of cameras andchroma-key billboards will be situated on another or the opposite sideto be viewed by a second plurality of cameras. This, for example, thehome nation may view the normal billboards, with the international TVaudience seeing only substituted boards.

The present invention also provides a method for electronicallyreplacing a billboard in a video image display, generated by a camera,comprising the steps of:

a. identification of the position of a rectangular billboard in astadium or other venue, said identification step comprising specifyingon the video display the billboard to be replaced by identification ofits four corners at a first camera position;

b. storing the identification information;

c. monitoring the movement of the camera in pan, tilt and zoom;

d. storing the monitored movement of the camera on a field by fieldbasis; and

e. analysing the size and position of the billboard to be replaced fromthe information recorded in its first known position and the storedmovements of the camera to provide information relating to the size,perspective and position of the billboard in the present video field;

f. storing in a billboard replacement store a replacement billboard tobe used in replacement of the billboard in the stadium;

g. electronically altering the size and perspective of the replacementbillboard in accordance with the camera motion information to conform tothe size and perspective of the billboard to be replaced in the presentvideo frame; and

h. electronically replacing the billboard in the present video frame bythe replacement billboard.

In a preferred embodiment the step of analysing the size and position ofthe billboard to be replaced comprises a further step of analysing aplurality of video scan lines to provide fine adjustment informationrelating to the exact size, perspective and position of the billboard tobe replaced.

In a further preferred embodiment the billboard to be replaced is blankand is of colour suitable for chroma-key replacement. Such colour may beof a blue or green shade, due to the fact that these colours are rarelyfound in human skin and hair.

In a still further preferred embodiment the chroma-key billboard ispatterned with a pattern of a suitable shape for the purpose offacilitating the fine adjustment process mentioned above. The step ofanalysis of the size and position of the billboard comprises theanalysis of the pattern to ascertain the exact position of thebillboard.

In a further embodiment the correction of the sensor-based prediction bythe analysis of the pattern will be controlled by a figure of merit(accuracy estimate) for the analysis, which will be computedautomatically.

In a further embodiment the step of electronically replacing thebillboard in the present video field by the replacement billboardinclude the step of superimposing occluding objects by use of thechroma-key techniques.

In a further preferred embodiment the billboard to be replaced can bechanged to best match the colours and shades of colours on the playerscostumes, for the purpose of providing a good contrast between thebillboards and the players. For example, if these costumes containshades of blue, then a green billboard may be selected.

The backing colour can be selected between blue, green and red. In orderfor the chroma-keyer to calculate all parameters necessary to performproper image compositing, the system requires a sample of the backgroundcolour as reference. This step can be done automatically by scanning theimage and detecting the purest and brightest colour. Advancedchroma-keyers enable the user to manually select the area to be sampled.

In a further preferred embodiment the chroma-key apparatus will have amultiplicity of set up conditions, each corresponding to a differentregion of the stadium. The camera pan, tilt and zoom information willallow to load the corresponding set up conditions.

In a further embodiment, the fine adjustment information will be used tocompensate drift errors of the sensors. In a practical situations, thesensor error will have a significant portion which is at temporalfrequencies which are much lower than the video field rate. Thus thesesensor-induced errors can be reliably estimated from good video fieldand subtracted from subsequent measurement.

The present invention also provides apparatus for carrying out themethod of electronically replacing the billboard as specifiedhereinbefore.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will now be described, by way ofexample with reference to the accompanying drawings in which:

FIG. 1 shows a stadium or other venue illustrating the apparatusaccording to the present invention;

FIG. 2 shows the video image of the stadium as seen by the camera in afirst position;

FIG. 3 illustrates a stadium with billboards in several differentpositions;

FIG. 4 illustrates a zoomed camera shot of a billboard illustrating theproblem with prior art systems;

FIG. 5 shows a patterned chroma-key billboard for use with the presentinvention;

FIG. 6 shows in block diagrammatic form circuitry associated with thecamera arrangement of FIG. 1 for transmitting video data and cameraorientation data;

FIG. 7 shows in block diagrammatic form receiver circuitry forco-operation with the transmitter circuitry of FIG. 6;

FIG. 8 shows a flow diagram for the operation of the circuitry of FIG.7;

FIG. 9 shows an arrangement for a billboard setup data store;

FIG. 10 shows an arrangement for billboard setup data;

FIG. 11 shows a flow diagram for perspective transformation computation;

FIG. 12 shows an arrangement for camera intrinsic parameters store;

FIG. 13 shows equations for dynamic recalibration;

FIG. 14 shows a flow diagram for dynamic recalibration;

FIG. 15 shows the process of recalibration;

FIG. 16 illustrates the problems relating to siting of billboards atdifferent locations within a stadium with differing lighting conditions;

FIG. 17 shows a graph of minimum and maximum levels for U & Villustrating the operation of a chroma-keyer;

FIG. 18 shows a billboard with occluding object illustrating theprinciple of adjustment of the chroma-key colour for a billboard;

FIG. 19 shows an exemplary remote receiver for reception of billboardcoordinate data and perfect chroma-key colour, occlusion being effectedby chroma-key techniques;

FIG. 20 shows an alternative arrangement for the billboard set up datastore illustrating an alternative embodiment;

FIG. 21 shows a flow diagram for dynamic set up procedure for camerapanning for use with the billboard set up data store of FIG. 20; and

FIG. 22 shows a flow diagram for dynamic set up procedure for cameratilting for use with the billboard set up data store of FIG. 20.

With reference now to FIGS. 1 to 4 the principle of the presentinvention is now explained.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In a stadium or other venue 10 billboards 14, 16, 18 are installed atthe side of a pitch represented by markings 12. These billboards areviewable by camera 20. Billboards 15,17,19 may be present on theopposite side of the stadium for viewing by a further camera 21. Thestadium terraces/seating are shown diagrammatically by the lines 11.

Camera 21 may in a preferred example be a normal TV video camera andwill transmit its output video signal directly to a first feed which mayserve the local population. Although we refer to camera 20 or 21, it maybe clearly understood that there could be a plurality of cameras on eachside of the stadium providing differing views.

Camera 21 in a preferred embodiment will televise boards 15,17,19 whichwill be transmitted to the local population in an unchanged manner.

Camera 20 will in this preferred embodiment transmit a feed to aninternational audience. Camera 20 is equipped with orientation sensingmeans which preferably comprises one or more of the following

pan measurement means 24;

tilt measurement means 25;

zoom measurement means 26; and

focus measurement means 28.

Suitable sensors may comprise the Virtual Reality Encoder from RADAMECEPO, Bridge Road, Chertsey, Surrey KT16, 8LJ, England.

Dependent on the allowed mobility of the camera only one, several or allof these may be required. For example, if camera 20 is fixed in pan andtilt and focus and can only zoom, as in the case of some remotelycontrolled unmanned cameras then only the zoom parameter need bemeasured.

Most cameras in sports stadiums can zoom, tilt and pan and it is assumedthat these parameters are measured for each camera as now explained. Thefocus is assumed to be fixed but in similar manner the parameter couldbe added if required.

FIG. 2 shows the video image as seen by a viewer and in particular bythe operator of the equipment. The camera 20 is zoomed, panned and/ortilted to “centre” the billboard 14 in a suitable position and at areasonable size. With reference to FIG. 7 each billboard is then viewedat a receiver and its position is marked preferably by using a touchscreen 700, or keyboard mouse 702 and marking the four corners. Thepositions are stored in a store 704.

For billboards higher in the stadium such as 30 (FIG. 3) a correctionfactor for the camera may be stored dependent on the tilt position ofthe camera.

Each billboard position is stored in store 704 together with the cameraparameter information at the reference position for the camera 20obtained from the camera parameter information which is correct at thetime that the billboard position is stored.

The following procedure is preferably repeated for each of the camerasand for each of the target billboard:

1. Point the camera at the target to obtain a stable unoccluded view ofthe target. Adjust the zoom to get a large view of the target yetkeeping the whole target within the field of view.

2. While the camera is not moving trigger an acquisition device, to graba picture of the target, as well as the corresponding readings of thesensors.

3. Mark the corners of the target, on the video image.

Preferably a corner detector is used to pinpoint the corners of thetarget at sub-pixel precision.

This camera parameter information is obtained (FIG. 6) from the sensorsmounted on the camera and the camera movement is referenced to a firstor fixed reference position for each parameter. The movements of thecamera are sensed and the signals are fed into a combiner circuit 34 andthen to a transmit buffer 36 from which the combined video and positiondata signals are transmitted.

During set up, at the receiver (FIG. 7) the receive buffer 706 receivesthe signals and feeds these to a splitter 708. The video signal isstored and delayed in a suitable store 710 and the camera parameter datais extracted and stored in store 712.

In set up the VDU 700 is used to mark each billboard that may requirereplacement. The camera 20 is panned etc to move each billboard into asuitable position on the screen and its position is recorded in thebillboard store 704 together with the camera parameters obtained fromstore 712 via processor 714.

A replacement billboard store 716 stores a plurality of replacementbillboards and these are selectable to be able to replace the originalbillboard.

The replacement billboard is in operation inserted into the video signalin a combiner 718 to provide a modified output video signal 720.

The setting up procedure can also identify billboard locations andcamera parameters for several cameras by storing a camera ID from asource 30 (FIG. 6). Thus, billboard position store 704 will storeseparate lists of billboard data for each camera.

The operation of the system will now be described with reference to asingle billboard and a single camera 20.

With reference to FIG. 4, it is assumed that billboard 14 enters thefield of view in an enlarged form on the left-hand side of the screen ascamera 20 pans following zooming from the FIG. 3 position.

The camera orientation data is constantly being received by the receiverand the processor 714 will constantly match on a pixel by pixel basisthe video image with the known billboard position stored in store 704.As soon as the billboard appears in the video image the pixelsrepresenting the billboard will be identified and the replacementbillboard pixels which relate to those pixels will be substituted in thecombiner 718. The delay will be minimal since the identification of thepixels is by an address correlation process which will be virtuallyinstantaneous.

After a period of time the camera sensors may drift and in this case thereplacement billboard may not exactly align with the original. This mayonly be by one or two pixels and may not be discernible to the viewer.To correct this two solutions are possible. Firstly, the billboardposition can be manually restored periodically at a suitable time, forexample, when a camera is not active. This requires the co-operation ofthe operator.

Secondly, a comparison can be made on a pixel by pixel basis of thebillboard against an original stored billboard and an adjustment of thereference camera parameters can be made in billboard position store 704.This process can be done automatically at either set intervals or whenthe processor 714 has a suitable time slot.

The essential steps of a preferred recalibration process are toperspectively transform the current video image using the camera data toprovide an estimated transformed model. A stored image of the billboardis then compared with the transformed model to provide a residual videofield. The residual distortion between the transformed model and theresidual video field is resolved to provide updating information forupdating the estimated transformation and to thereby provide acalibration correction factor for recalibrating the position of eachbillboard in the store in accordance with the camera sense information.

The replacement of each billboard is accomplished by use of theprocessor 714 (FIG. 7) and the various parameter and billboard storesusing appropriate software programmes as now described in more detail.

FIG. 8 describes the complete process which allows to determine theposition of each billboard in the camera's field of view, and render thecorresponding part of the billboard into the frame buffer. Since therendering and later the compositing of the graphics buffer with thevideo buffer by means of chroma-key are known art, we will concentrateon the billboard position determination with reference also to FIGS. 6and 7.

At the beginning of each video field, the pan, tilt, zoom and focussensors (24,25,26) are read 800. These values, combined with billboarddata from billboard setup data store 704 and camera data from cameraintrinsic parameters store 712, enable the detection and recognition ofall billboards in the camera's FOV, independently of the video signal.The processing of FIG. 1 consists of a loop on all billboards (m)802,804. For each billboard, its setup data is retrieved 806 frombillboard setup data store 704 and used with camera intrinsic parameters808 to compute the perspective transformation 810 from billboard m tocurrent field. The replacement billboard information is then stored(812) in a frame buffer.

FIG. 9 describes the billboard setup data store 900 which consists of aseparate record 902 . . . 904 for each billboard in the arena. Such arecord consists of a static image 906 grabbed in favourable conditionsand of the corresponding static setup data 908. The record also consistof dynamic setup data 910 which is computed using the image processingmeans in a process known as dynamic re-calibration which has beenbriefly described above and will be further described with reference toFIG. 11. An alternative procedure providing static and dynamiccalibration is described with reference to FIGS. 20, 21 and 22.

FIG. 10 describes the setup data (either static or dynamic) 1000 for asingle billboard. It consists of the sensors' readings 1002 at the setupinstance, the billboard quadrilateral vertex 1004 coordinates and thetime-code of the setup instance 1006.

The method of dynamic re-calibration can be explained as follows:

Due to sensors' drift and inaccuracies, a final calibration table andother practical reasons, it is impossible to predict the exact locationof all visible billboards at a given instance. However, at many videofields, a billboard's visibility may be such that an exact geometricposition correction can be performed. Since that position is closer bothtemporally and spatially to the subsequent video fields, it ispreferable to rely on that “luck shot” by predicting the billboardposition relative to its sensors' readings and exact quad coordinates.Consider for example a billboard which exit s the field of view due tocamera panning. Having a luck shot while it is still highly visible,allows the smooth tracking of the billboard by sensors only, when itsvisibility does not allow any image processing means to be applied.

FIG. 11 depicts the flow diagram 1100 for perspective transformationcomputation. A setup data selection logic 1102 selects either the static1103 or the dynamic 1105 setup data from the setup data store 806 asdescribed above. This setup data, together with camera intrinsicparameter is use to compute a sensor-based prediction of the perspectivetransformation 1104, independently of the video signal.

A dynamic re-calibration 1106 based on image processing means is thenapplied to the prediction. It utilizes the video 1108 and chroma-key1110 signals as well as the billboards model image 1112 from the setupdata store 806 (FIG. 8). Based on a quality factor derived from theimage processing means, either the sensors-based 1118 or the correctedtransformation 1116 are output. If the estimated quality of thegeometric correction is high, then the dynamic setup data is updated1114.

FIGS. 12,13,14 describe the sensor-based prediction of billboardcoordinates in the video field. Such a prediction utilizes the sensorsreading as well as the camera intrinsic parameters. These parameters aredescribed in FIG. 12 and have to be tabulated for a dense sampling ofthe (zoom,focus) space. The meaning of these parameters is clear fromFIG. 13 to which reference is now made.

Let the set of measurements given by the pan, tilt, zoom and focussensors be represented by the vector (P,T,Z,F). The tilt angle isassumed to be relative to the horizon.

Consider an object point whose image at some setup instance is, atframe-buffer coordinates (x_(s), y_(s)). Let also the sensor measurementvector at that instance be (P_(s), T_(s), Z_(s), F_(s)).

At another instance, the prediction instance, let the sensormeasurements vector be (P_(p), T_(p), Z_(p), F_(p)). It is required topredict the location of the object point in frame-buffer coordinates(possibly out of the actual frame-buffer), (x_(p), y_(p)).

To enable the procedure we define the setup rotation matrix as shown at600 and the prediction rotation matrix is defined as shown at 602.

Then, the perspective transformation matrix between the two image-planecoordinate systems is given as shown at 604 and 1402 (FIG. 14).

R_(sp) is a 3*3 matrix with row and column indices ranging from 0 to 2.R_(sp)[i][j] denotes the term in row i, column j in the matrix. Thus,given the setup image-plane coordinates of the object point (u_(s),v_(s)), the predicted location of the object point in image-planecoordinates (u_(p), v_(p)) is given as shown at 606, 1404.

Image-plane to frame-buffer coordinate transformation is achieved asshown at 608, 1406. Aberration compensation is achieved as shown at 608,1406 (FIG. 14) to provide predicted frame buffer billboard coordinatesand perspective transformation data.

An effective way of deriving these parameters for a specific(zoom,focus) pair is described in [J. Weng et al., Calibration of stereocameras using a non-linear distortion model, IEEE 10th Intl. Conf.Pattern Recognition (1990), pp. 246-253]. The image processing means forgeometric correction, which allow also the process of re-calibration, isnow described with reference to FIG. 15.

The image processing means for geometric correction of sensors-basedprediction are based on the differential method for motion estimation[C. Cafforio and F. Roca, The differential method for motion estimation,in: T. S. Huang, eg., Image sequence processing and dynamic sceneanalysis, Spring, Berlin, 1983, pp. 104-124]. Let C be the current videofield and let M be the static billboard setup image, perspectivelytransformed according to the sensors-based prediction. Here we consideronly luminance images. Ideally, M and C are identical within the supportof the billboard quadrilateral. Actual differences may include:

Occlusion present in C but not in M.

Geometric errors due to sensors and intrinsic camera parameters errors.

Luminance changes.

Neglecting for the moment any difference which is not due to geometricerrors, consider a point (x,y) inside the support of the billboardquadrilateral. Let (p,q) be the local geometric error then we may writefor the luminance signals of the respective images:

M(x+p,y+q)=C(x,y)

Under the assumption that the error is small, one may write a Tailorseries expansion:${M\left( {{x + p},{y + q}} \right)} = {{M\left( {x,y} \right)} + {p\frac{M}{x}} + {q\frac{M}{y}} + \left( {{Second}\quad {order}\quad {terms}} \right)}$

Neglecting the second order terms and denoting the spatial derivatives

dM/dx=H

dM/dy=V

we obtain

C(x,y)−M(x,y)=pH+qV

Also denoting the differences C(x,y)−M(x,y) by D we obtain

D=pH+qV

The equation above holds, locally. For a global billboard solution, andsmall error assumption we may use the perspective model [G. Adiv,Determining Three-Dimensional Motion and Structure from Optical FlowGenerated by several moving objects, IEEE Trans. Pattern Analysis andMachine intelligence, 7, pp. 384-401, 1985]. $\begin{matrix}{{p\left( {x,y} \right)} = \frac{{a_{1}x} + {a_{2}y} + a_{3}}{{a_{7}x} + {a_{8}y} + 1}} \\{{q\left( {x,y} \right)} = \frac{{a_{4}x} + {a_{5}y} + a_{6}}{{a_{7}x} + {a_{8}y} + 1}}\end{matrix}$

The coefficients a1 , . . , a8 are computed by minimizing the followingexpression:$\sum\limits_{({x,y})}\quad \left( {{D\left( {x,y} \right)} - {{p\left( {x,y} \right)}{H\left( {x,y} \right)}} - {{q\left( {x,y} \right)}{V\left( {x,y} \right)}}} \right)^{2}$

Now, the perspective transformation matrix (based on sensors'prediction) is multiplied by: $\begin{bmatrix}a_{1} & a_{4} & a_{7} \\a_{2} & a_{5} & a_{8} \\a_{3} & a_{6} & 1\end{bmatrix}$

The matrix obtained can be considered to be the updated prediction ofbillboard perspective. In a practical environment the followingconsiderations may apply.

Occlusion may cause major problem in this formulation, since if pixelsfrom occluding and moving objects participate in the minimization of theexpression above they might bias the solution significantly. Preferably,such pixels are discarded from processing by using chroma-key panels. Akey signal output by a chroma-keyer, is preferably utilized to discardthese pixels.

Luminance variations can be minimized by pre-processing the currentvideo field, using histogram matching techniques.

The prediction-correction process may require 2-3 iterations toconverge.

Noise immunity and convergence can both be enhanced by pre-smoothing theimages.

Thus the billboards 14 etc are in accordance with the present inventionchroma-key boards and occlusion is by colour discrimination using thenormal chroma-key techniques. These techniques will enable perfectocclusion providing that the players do not wear any colour which is thesame as the board. This may not always be possible and it is proposed inaccordance with a particular embodiment of the present invention to useboards which can rotate or otherwise change to a second or third colour.For example, three colours may be blue, green and red which may beselected when the colour of the players strips are known.

Alternatively if it is required to display a billboard in an area of thepitch or a surrounding area then such an area must be selected to be ofa known colour which can then be recorded in the chroma-keyer as achroma-key colour.

In a preferred embodiment chroma-key apparatus can comprise theULTIMATTE-7 digital video image compositing device from ULTIMATTE Corp.,20554 Plummer St., Chatsworth, Calif. 91311, USA.

The backing colour can be selected between blue, green and red. In orderfor the chroma-keyer to calculate all parameters necessary to performproper image compositing, the system requires a sample of the backgroundcolour as reference. This step can be done automatically by scanning theimage and detecting the purest and brightest colour. Advancedchroma-keyers enable the user to manually select the area to be sampled.

In a particular embodiment it is proposed to use a patterned chroma-keypanel. Calibration of the camera sensors can then readily beaccomplished by comparison of the pattern on a pixel by pixel basis. Thepattern on the billboard panel should preferably have criticaldimensions less than anticipated sensors error (projected to worldcoordinates).

In summary, the above system can operate even in extremely poor weatherconditions since the electronic processing circuitry knows exactly whereeach billboard is situated and does not rely on any analysis of thevideo image to detect the billboard. In the event that the video imageis so distorted that recalibration cannot be carried out with reasonablecertainty, then the original camera parameter settings can continue tobe used since the video image as viewed will be of poor quality and thusthe viewer will not notice an error of one or two pixels in thepositioning of the replacement billboard which will require to bedisplayed in an equivalent quality which matches the poor quality videoimage.

In a further preferred embodiment of the present invention, the problemaddressed is that of having billboards situated in different positionsin a stadium as shown in FIG. 16.

In such conditions the lighting of billboards, 1,3 and 5 will bedifferent because of the location of lights 7, 9 and 13. Also thislighting can change all the time during the game.

If such billboards are chroma-key boards all of the same colour then thebillboards will all appear to be slightly different colours due to thedifferent lighting conditions.

A fixed adjustment of a global backing colour might result in partialobject background separation by the chroma-keyer.

In the present invention it is proposed to provide spatial adaptation ofthe backing colour map so that the chroma-keyer can correctly recogniseeach billboard. This can be provided by storing in store 704 (FIG. 7) aspatial map providing information relating to the colour of each chromakey board.

Thus, the chroma-keyer will compare the colour in each video locationwith a specific colour associated with the billboard in that location.

In a preferred embodiment the locations of the billboards may beidentified by “painting” a slightly enlarged box surrounding thebillboard to identify the location. Such boxes are identified as 1′, 3′and 5″ by dotted lines in FIG. 16.

The system will track the backing colours over time and therefore willcontinuously update, to ensure correct identification, once correctlyset up.

The operation of the system is as follows.

Firstly with reference to FIG. 17, minimum and maximum levels are setfor U and V. These should be wide enough to encompass all billboardswhich are reasonably lit.

Then for each billboard, as lighting conditions change, an adjustmentcan be made to its stored values, as shown in FIG. 18, which assumesocclusion of billboard 1 by an object 13. An inner box 1″ is defined toensure only pixels from within 1 are considered. Most occluding pixelscan be discarded as these will be of a different colour. Then all pixels(YuV) inside FOV and billboard quad 1″ are measured and an addition tothe average (which is backing colour average UV over billboard) is madeif:

U_(min)≦U≧U_(max) or

V_(min)≦V≧V_(max)

The inventors have recognised a further problem which arises from theuse of chroma-key billboards in a stadium. Due to the variable lightingas described above, each billboard will appear on the video image as aslightly different colour. In order to transmit correct occlusioninformation it is necessary to transmit an occlusion map for eachbillboard.

In accordance with a preferred embodiment of the present invention it isproposed to transmit, for each billboard, a perfect background colourand to then allow a chroma-keyer in each receiving station to introducethe occluded portions by normal chroma-key procedures.

Consider now the billboard arrangement as shown in FIG. 16. Eachbillboard 1,3 and 5 will, because of its different lighting conditions,appear to be a different colour even though this colour may be withinthe maximum and minimum limits as set out in FIG. 17.

In accordance with this preferred embodiment of the present invention,the transmitting apparatus (see FIG. 6) will transmit a perfectchroma-key colour within the area of the billboard and will alsotransmit the coordinates of the quadrilateral formed by the billboard.

In this way the receiving station only has to decode/extract thequadrilateral coordinates of the billboard and then within thatquadrilateral replace those pixels which are the perfect chroma-keycolour by the replacement billboard. Those pixels which are not aperfect chroma-key colour are not replaced.

In accordance with this system it is not necessary for the chroma keyerat the remote receiving station to be able to recognise differentbillboards and to have to store different chroma-key values for eachbillboard. Also it is not necessary to transmit any occlusioninformation since occlusion by the chroma-keyer will be relativelysimple at each remote location.

With reference to FIG. 7, the billboard position and backing colourstore 704 knows the position of each billboard and a control output 7042from the store is used, in combination with the video output 7102 toprovide inputs for a backing colour processor 7044 which can change thecolour of the billboard within the coordinates provided by store 704.The output of processor 7044 is used to control a backing colour store7046 which changes the colour of the billboard within the requiredcoordinates and can also provide the coordinates to the video output 720for the remote receiver. These may be transmitted by a standard videodata transmission system.

An exemplary remote receiver is shown in FIG. 19. Video data is receivedat receiver buffer 1900 and split and delayed 1902, 1904.

Billboard coordinate store 1906 stores the transmitted billboardcoordinates and in combination with graphics generator 1908 andreplacement billboard image store 1910 provides an output signal to acombiner/chroma-keyer 1912 to produce the desired, occluded billboard onthe screen.

With reference now to FIGS. 20 to 22 in a further embodiment the set updata stored in store 704 is modified prior to any event being televised.

The modification comprises the addition of dynamic set up data as wellas the static image and static set up data shown in FIG. 9.

The additional data may be used instead of the dynamic recalibration setup data 910 shown in FIG. 9 or could be used in addition.

In a preferred embodiment it is assumed that the additional data is usedinstead of the dynamic recalibration procedure and this is nowdescribed.

As an introduction, the problems associated with replacing billboardswith virtual billboards are discussed. The same problem is identifyingthe position, size and perspective of the original billboard and thenreplacing this with the virtual or replacement billboard.

In a static camera situation there is no real problem once the originalco-ordinates have been recorded providing that the camera sensors do notdrift substantially over time.

However, the inventors have found that during rapid panning or tiltingof the camera the co-ordinates 908 of the replacement billboard asrecorded in the store 900 do not coincide with the actual position ofthe billboard in the stadium or venue. This is because the camerasensors exhibit a degree of hysteresis. This can be compensated for bythe dynamic recalibration process already described but this may not bepractical in some circumstances such as during rapid panning withsubstantial occlusion of the target billboard.

The hysteresis could possibly be countered by a simple percentage errorbuilt into the movement of the camera but this does not produce verygood results because it does not take into consideration the cameraangle with respect to each billboard nor does it take into account thevariability in the camera parameter sensors with angle.

In the present invention therefore in an alternative embodiment, inaddition to static billboard set up data the billboard set up data store900 stores data for each billboard for each camera at least in relationto left-right pan 2002, right-left pan 2004, up-down tilt 2008 anddown-up tilt 2006. The data is obtained and stored as now hereindescribed with reference to FIGS. 20 to 22.

FIG. 20 shows the store 900 modified to provide, in addition to thestatic image data for billboards 1 to M and the static set up data forbillboards 1 to M four further sets of data for each billboard 1 to Mand these are multiplied to provide this data for each camera.

For each camera the position of each billboard is recorded with thecamera panning from left to right 2002 and for right to left 2004. Thepanning speed may be selected as the normal speed for the event beingtelevised. Thus for example for horse racing it could be low but formotor racing it could be higher. The position of each billboard is thenrecorded with the camera tilting upwards 2006 and then downwards 2008across each billboard.

For each measurement the camera zoom and focus are preferably set at aknown level at which the billboard being analysed is in a reasonableview at a reasonable size. The zoom and focus could for example be thesame as that during the acquisition of the static set up data for eachbillboard so that a direct comparison with the static set up data can bemade. In that case only an error correction figure may need to berecorded.

It is preferable not to select too high a panning or tilting speedbecause at very high speeds the billboards will in any case be blurredand therefore accuracy of replacement will not be an issue. Thisprocedure is followed for each billboard for each camera and the data isthen used during the event to correct the position of each billboard asthe camera pans or tilts.

It may be seen that each billboard will be viewed at a different angleby each camera and also that the output of each of the sensors on eachcamera may vary dependent on the angle through which the camera mustturn to view the billboard. By recording the static data and datarelating to panning and tilting in both directions the replacementbillboard will be accurately positioned in the exact position of theoriginal or real billboard both for static shots and when the camera ismoving.

Dynamic recalibration during the event as previously described willensure, except during very fast camera movements with large occlusion,that the replacement billboard is correctly positioned but the use ofstatic and dynamic set up data will also ensure this unless the camerasensors drift substantially during an event. Thus providing that thecamera sensors are of a reasonable quality from the point of view ofdraft they can be of a variable quality with respect to accuracy duringpanning and tilting. By careful selection of camera sensors extremelyaccurate sensors are therefore not required since any variation withrespect to camera movement is compensated for by the storage of dynamicset up data.

The data is obtained as described with reference to FIGS. 21 and 22 asfollows.

Once the static data relating to each billboard image and its static setup data (906, 908, FIG. 9) has been obtained, sequence 2100 is startedand the camera is panned 2102 by the operator at a desired speedrelative to the normal panning speed.

The sensors indicate the direction of pan 2106 and dependent on thedirection the dynamic data is stored in store 2002 or 2004 in steps 2106or 2108 by selection of that store. The sequence for both L-R and R-Lstores 2002 and 2004 is similar and will be described for the L-R storebut using reference numerals for both stores.

As the camera pans L-R each billboard is identified from data stored instore 906 step 2110; 2112. The system asks if the billboard haspreviously been recoded dynamically step 2114, 2116 and if so it returnsto the start of the sequence and repeats steps 2104-2110 until it findsa billboard that has not been dynamically scanned. Once a new billboardhas been found the position (co-ordinates) of the billboard duringpanning is recorded and compared with the static billboard parameterspreviously stored (908) in step 2118, 2120. Any error is computed (step2122, 2124) and the errors are stored in L-R and R-L pan stores 2126,2128 for the billboard. The system asks if all billboards recorded instore 908 have been dynamically scanned both for L-R and R-L (steps2130, 2132). If not the sequence is continued until the last billboardhas been dynamically scanned and then the program is terminated 2134,2136.

A similar program sequence shown in FIG. 22 is provided for tilting ofeach camera. Obviously if the cameras are not either allowed to tilt orare unlikely to be tilted to any extent then this sequence and therecordal of data in stores 2006, 2008 may not be necessary.

The sequence is started 2200 and each camera in turn is tilted 2202 andthe direction of tilt determined 2204 by the camera sensors. Dependenton whether the camera is tilting up or down dynamic set up data isstored in stores 2006 or 2008 in steps 2206, 2208. Both sequences aresimilar and only the sequence tilting the camera down will be describedwith reference then to both sequences.

Each billboard is identified 2210, 2212 from the static image data andalso from the camera parameters especially where all real billboards arethe same. The program interrogates the billboard data step 2214, 2216 tosee if the billboard has already been interrogated. If it has theprogram restarts but if not the co-ordinate data of the billboard duringtilting is compared with the static data step 2218, 2220. The error, ifany is computed step 2122, 2124 and stored in the stores 2006, 2008(FIG. 20) step 2226, 2228.

The program then interrogates stores 2006, 2008 to see if all billboardshave been dynamically interrogated for tilt errors in both up (step2230) and down (step 2232) and if so ends the program steps 2234, 2236.If not the program continues by commencing at the start of the sequenceuntil all boards have been interrogated.

Normally camera zoom and focus will not require the same type of dynamicset up data to be stored. However, if particular camera aberrations areknown then these may be compensated for by use of similar dynamic set updata.

The dynamic data stored in stores 2002-2008 may be used instead of or inconjunction with the dynamic recalibration data obtained as describedwith reference to FIG. 9. Usually however the dynamic set up data willobviate the need for recalibration during most types of event.

During use the system knows by reading the camera sensors whether thebillboard is being viewed in a static manner or is being panned past L-Ror R-L or tilted past UP or DOWN. In such cases the position of thebillboard is taken from the static data store and then if panning ortilting is occurring, the necessary error corrections are applied. Oncecamera movement ceases the static billboard parameters are reverted to.

What is claimed is:
 1. Apparatus for automatic replacement of abillboard in a video image, comprising: (i) a camera position store thatreceives input from sensors of a camera that are indicative of aposition of the camera and that stores the input; (ii) a billboardposition store that stores a position of the billboard; (iii) aprocessor that determines from the stored input in the camera positionstore and the stored position of the billboard in the billboard positionstore and independently of a video signal generated by the camerawhether the billboard is within the field of view of the camera; and(iv) a replacement billboard store that stores a replacement billboardfor replacing the billboard when the processor determines that thebillboard is within the field of view of the camera.
 2. Apparatus forautomatic replacement of a billboard in a video image as claimed inclaim 1, further comprising a calibration means to periodicallyautomatically calibrate the sensors.
 3. Apparatus as claimed in claim 1,wherein the sensors comprise means for measuring the pan tilt, zoom andfocus of the camera relative to known reference positions.
 4. Apparatusas claimed in claim 2, wherein the sensors comprise means for measuringthe pan tilt, zoom and focus of the camera relative to known referencepositions.
 5. Apparatus as claimed in claim 2, wherein the billboardcomprises a chroma-key panel.
 6. Apparatus as claimed in claim 5,wherein the panel comprises a patterned chroma-key panel.
 7. Apparatusas claimed in claim 6, wherein the pattern comprises two differentcolors or two different shades of the same color.
 8. Apparatus asclaimed in claim 7, wherein the pattern is designed to be optimized foruse according to the size and shape of the billboard and to theanticipated video conditions.
 9. Apparatus as claimed in claim 2,wherein the calibration means includes fine adjustment image processingmeans including means for perspectively transforming the current videoimage to provide an estimated transformed model, means for storing areplica of the billboard image the perspectively transformed video imagewith the stored replica to provide a residual video field, means forsolving residual distortion between the transformed model and theresidual video field and means for updating the estimated transformationby means of the residual distortion to provide a calibration correctionfactor.
 10. Apparatus as claimed in claim 9, wherein said means forsolving residual distortion includes analysis means based in the spatialderivatives of the current image of the transformed model as well asmeans for analyzing pixel by pixel the image differences of thetransformed model and the current image.
 11. Apparatus as claimed inclaim 9, wherein the fine adjustment image processing means includesmeans for carrying out the fine adjustment in an interactive manner. 12.Apparatus as claimed in claim 10, wherein the pixel by pixel analysismeans includes means for discarding pixels using color variation. 13.Apparatus as claimed in claim 1, further comprising means for recordingcalibration data obtained during a set up period when no occlusion ispresent.
 14. Apparatus as claimed in claim 1 further comprising adynamic billboard memory for recording any changes in measured positionof the billboard as the camera pans or tilts.
 15. An apparatus forautomatic measurement of camera orientation, comprising: (i) a sensormeans for measuring a Field of View (FOV) of a camera relative to aknown reference position; (ii) an image processing means for processingvideo signals generated by the camera; and (iii) a calibration means forperiodically calibrating the sensor means, wherein the calibration meansperspectively transforms a current camera image to generate anestminated transformed model, compares a stored image with the estimatedtransformed model to form a residual video field, and determinesresidual distortion for calibration by determining residual distortionbetween the estimated transformed model and the residual video field.16. A method of automatic replacement of a real billboard in a videoimage said method comprising the steps of: recording the Field of View(FOV) parameters of a TV camera in a first reference position by meansof motion sensors attached to the camera; storing a position of the realbillboard in a billboard store; storing a position of the camera sensedby the motion sensors in a camera position store; determining from thecamera position store and the stored position in the billboard store andindependently of a video signal generated by the camera whether the realbillboard is within the field of view of the camera; and replacing thereal billboard with a replacement billboard when the processordetermines that the real billboard is within the field of view of thecamera.
 17. The method as claimed in claim 16, wherein storing aposition of the camera comprises measuring the pan, tilt, zoom or focusof the camera with the motion sensors attached to the camera.
 18. Themethod of claim 16 wherein the billboard comprises a chroma-key paneland in which replacement of the real billboard with the replacementbillboard includes the step of occluding said replacement billboard byany object occluding said real billboard.
 19. The method of claim 16further comprising periodically automatically calibrating the motionsensors by perspectively transforming the current video image to providean estimated transformed model, storing a replica of the billboardimage, comparing the perspectively transformed video image with thestored replica to provide a residual video field, solving residualdistortion between the transformed model and the residual video fieldand updating the estimated transformation by means of the residualdistortion to provide a calibration correction factor.
 20. The method ofclaim 16, wherein said step of solving the residual distortion includesanalysis based in the spatial derivatives of the current image of thetransformed model and analyzing pixel by pixel the image differences ofthe transformed model and the current image.
 21. The method of claim 16,wherein automatically calibrating the motion sensors further comprisesinteractively calibrating the motion sensors.
 22. The method of claim20, wherein the pixel by pixel analysis step includes discarding pixelsby using color variation.
 23. The method of claim 16, wherein the stepof recording calibration data is carried out during a set up period whenno occlusion is present.
 24. The method of claim 16, further comprisingcalibrating a difference signal for recalibration of the motion sensorswhen the present video image is above a predetermined quality factor.25. The method of claim 24, wherein the replacement billboard ispositioned at the last recalibrated position in the event that arecalibration is not possible due to the present video image being belowthe predetermined quality factor.